<?xml version="1.0" encoding="UTF-8"?>
<!-- -*- coding: utf-8 -*- -->

<!DOCTYPE html PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN"
	  "http://www.w3.org/TR/html4/loose.dtd">

<html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en">

<head>
  <!-- meta http-equiv="Content-Type" content="text/html; charset=utf-8" -->
  <link rel="stylesheet" type="text/css" href="style/style.css" />

  <title>Installing HPCToolkit with Spack</title>
</head>

<body>

<!-- HPCToolkit Image (height = 71 pixels) -->
<img style="position: absolute; top: 0px; left: 0px" src="style/header.gif" alt="hpctoolkit" />
<div style="position: relative; margin-top: 60px; width: 100%; text-align: right;">
<p style="margin: 0px; font-size: small;">
  [ <a href="index.html">Home</a>
  | <a href="overview.html">Overview</a>
  | <a href="publications.html">Publications</a> ]
  &bull;
  [ <a href="examples.html">Examples</a> ]
  &bull;
  [ <a href="software.html">Software/Downloads</a>
  | <a href="documentation.html">Documentation/Questions</a> ]
  &bull;
  [ <a href="info-people.html">People</a> |
    <a href="info-acks.html">Acks</a> ]
</p>
</div>
<div style="width: 100%;">
  <hr />
</div>

<h1 class="settitle" align="center">Installing HPCToolkit with Spack</h1>

<a name="SEC_Contents"></a>
<h2 class="contents-heading">Table of Contents</h2>

<div class="contents">
<ul class="no-bullet">
<li><a name="toc-Introduction" href="#Introduction">1 Introduction</a></li>
<li><a name="toc-Prerequisites" href="#Prerequisites">2 Prerequisites</a></li>
<li><a name="toc-Clone-Spack-and-HPCToolkit" href="#Clone-Spack-and-HPCToolkit">3 Clone Spack and HPCToolkit</a></li>
<li><a name="toc-Config_002eyaml" href="#Config_002eyaml">4 Config.yaml</a></li>
<li><a name="toc-Packages_002eyaml" href="#Packages_002eyaml">5 Packages.yaml</a></li>
<li><a name="toc-Compilers-and-compilers_002eyaml" href="#Compilers-and-compilers_002eyaml">6 Compilers and compilers.yaml</a></li>
<li><a name="toc-Spack-Install" href="#Spack-Install">7 Spack Install</a></li>
<li><a name="toc-Advanced-Options" href="#Advanced-Options">8 Advanced Options</a>
<ul class="no-bullet">
  <li><a name="toc-MPI" href="#MPI">8.1 MPI</a></li>
  <li><a name="toc-PAPI-vs-Perfmon" href="#PAPI-vs-Perfmon">8.2 PAPI vs Perfmon</a></li>
</ul></li>
<li><a name="toc-Platform-Specific-Notes" href="#Platform-Specific-Notes">9 Platform Specific Notes</a>
<ul class="no-bullet">
  <li><a name="toc-Blue-Gene" href="#Blue-Gene">9.1 Blue Gene</a></li>
  <li><a name="toc-Cray" href="#Cray">9.2 Cray</a></li>
</ul></li>
<li><a name="toc-Hpcviewer-and-Hpctraceviewer" href="#Hpcviewer-and-Hpctraceviewer">10 Hpcviewer and Hpctraceviewer</a>
<ul class="no-bullet">
  <li><a name="toc-Spack-Install-1" href="#Spack-Install-1">10.1 Spack Install</a></li>
  <li><a name="toc-Manual-Install" href="#Manual-Install">10.2 Manual Install</a></li>
</ul></li>
<li><a name="toc-Common-Problems" href="#Common-Problems">11 Common Problems</a>
<ul class="no-bullet">
  <li><a name="toc-Unable-to-fetch-tar-file" href="#Unable-to-fetch-tar-file">11.1 Unable to fetch tar file</a></li>
  <li><a name="toc-New-releases-break-the-build" href="#New-releases-break-the-build">11.2 New releases break the build</a></li>
</ul></li>

</ul>
</div>


<a name="Introduction"></a>
<h3 class="section">1 Introduction</h3>

<p>These notes describe how to build and install HPCToolkit and hpcviewer
and their prerequisites with Spack.  HPCToolkit proper (hpcrun,
hpcstruct and hpcprof) is used to measure and analyze an application&rsquo;s
performance and then produce a database for hpcviewer.  HPCToolkit is
supported on the following platforms.
</p>
<ol>
<li> Linux (64-bit) on x86_64, big and little-endian powerpc (power7, 8 and
9) and ARM (aarch64).

</li><li> Cray XT/XE/XK/XC on x86_64 and Compute Node Linux.

</li><li> IBM Blue Gene/Q on ppc64.
</li></ol>

<p>We provide binary distributions for hpcviewer and hpctraceviewer on
Linux (x86_64 and powerpc), Windows and MacOS.  HPCToolkit databases are
platform-independent and it is common to run hpcrun on one machine and
then view the results on another machine.
</p>
<p>We build HPCToolkit and its prerequisite libraries from source.
HPCToolkit has some 15-20 prerequisites and we now use spack to build
them.  It is possible to use spack to install all of hpctoolkit or
build just the prerequisites and then build hpctoolkit with the
traditional <code>configure ; make ; make install</code> method from
autotools.  Developers will probably want to run <code>configure</code> and
<code>make</code> manually, but both methods are supported.
</p>
<p>Note: the old method of using hpctoolkit-externals to build the
prerequisite libraries is now superseded by spack.  Although externals
will continue to work for a while, we have stopped updating externals
and now add newer versions of packages only in spack.
</p>
<p>These notes are written mostly from the view of using spack to build
hpctoolkit and its dependencies.  If you are a more experienced spack
user, especially if you want to use spack to build hpctoolkit plus
several other packages, then you will want to adapt these directions to
your own needs.
</p>
<p>Spack documentation is available at:
</p>
<div class="example">
<pre class="example"><a href="https://spack.readthedocs.io/en/latest/index.html">https://spack.readthedocs.io/en/latest/index.html</a>
</pre></div>

<p>The current status of using Spack for HPCToolkit is at:
</p>
<div class="example">
<pre class="example"><a href="http://hpctoolkit.org/spack-issues.html">http://hpctoolkit.org/spack-issues.html</a>
</pre></div>


<a name="Prerequisites"></a>
<h3 class="section">2 Prerequisites</h3>

<p>Building HPCToolkit requires the following prerequisites.
</p>
<ol>
<li> The GNU gcc and g++ compilers version 4.8 or later and preferably 5.x or
later.  On systems with older compilers, you can use spack to build a
later version of gcc.

</li><li> Basic build tools: make, ld, ar, objcopy, nm, etc, and shell utilities:
bash, sed, awk, grep, etc.  Most Linux systems have these tools, or else
you couldn&rsquo;t compile anything.

</li><li> Cmake version 3.4 or later, perl version 5.x, and python version 2.7 or
3.4 or later.  On systems that are missing these tools or have versions
that are too old, you can use spack to build a later version.

</li><li> Git and curl for downloading files.

</li><li> (optional) Environment (TCL) or LUA (Lmod) modules, if you want to make
HPCToolkit available as a module.  Again, spack can install these
packages if they are missing from your system.
</li></ol>

<p>Hpcviewer and hpctraceviewer require Java version 8.  On Linux, the
viewers also require GTK+ version 2.x.
</p>

<a name="Clone-Spack-and-HPCToolkit"></a>
<h3 class="section">3 Clone Spack and HPCToolkit</h3>

<p>Spack is available via git clone from GitHub.  This includes the core
spack machinery and recipes for building more than 3,000 packages (and
growing).  You should also clone HPCToolkit for the
<code>packages.yaml</code> file which is used to configure the spack build.
</p>
<div class="example">
<pre class="example">git clone https://github.com/spack/spack.git
git clone https://github.com/hpctoolkit/hpctoolkit.git
</pre></div>

<p>After cloning, add the <code>spack/bin</code> directory to your PATH, or else
source the spack <code>setup-env</code> script.
</p>
<div class="example">
<pre class="example">(bash)   .  /path/to/spack/share/spack/setup-env.sh
(csh)    setenv SPACK_ROOT /path/to/spack/root
         source $SPACK_ROOT/share/spack/setup-env.csh
</pre></div>

<p>It suffices to add <code>spack/bin</code> to your PATH (or even symlink the
spack launch script).  Sourcing the <code>setup-env</code> script adds extra
support for modules built by spack.
</p>

<a name="Config_002eyaml"></a>
<h3 class="section">4 Config.yaml</h3>

<p><code>config.yaml</code> is the top-level spack config file.  This specifies
the directory layout for installed files and the top-level spack
parameters.  There are two or three fields in this file that you should
set for your local system.
</p>
<ol>
<li> <code>install_tree</code> &ndash; by default, spack installs packages inside the
spack repository at <code>spack/opt/spack</code>.  If you want to use another
location, then set this.

</li><li> <code>module_roots</code> &ndash; by default, spack installs module files inside
the spack repository at <code>spack/share/spack</code>.  If you want to use
another location, then set this.

</li><li> <code>build_jobs</code> &ndash; by default, spack uses all available hardware
threads for parallel make.  For machines with many threads, this can
sometimes overwhelm the system and you should set this to a smaller
number.  (There is precedent.)
</li></ol>

<p>There are also parameters for the locations of the build directories,
the cache of downloaded tar files, etc, which you may wish to set.
</p>
<p>The default <code>config.yaml</code> file is in the spack repository at
<code>spack/etc/spack/defaults</code>.  The simplest solution is to copy this
file one directory up and then edit the copy (don&rsquo;t edit the default
file directly).
</p>
<div class="example">
<pre class="example">cd spack/etc/spack
cp defaults/config.yaml .
vi config.yaml
</pre></div>

<p>Alternatively, you could put this file in a separate directory, outside
of the spack repository and then use <code>-C dir</code> on the spack command
line.  (The <code>-C</code> option goes before the spack command name.)
</p>
<div class="example">
<pre class="example">spack -C dir install ...
</pre></div>

<p>Note: if you put <code>config.yaml</code> in <code>spack/etc/spack</code>, then it
will apply to every spack command for that repository (and you won&rsquo;t
forget).  Putting it in a separate directory is more flexible because
you can support multiple configurations from the same repository.  But
then you must use <code>-C dir</code> with every spack command or else you
will get inconsistent results.
</p>
<p>You can view the current configuration with <code>spack config</code>.
</p>
<div class="example">
<pre class="example">spack [-C dir] config get config
</pre></div>

<p>See the spack docs on &lsquo;Configuration Files&rsquo; and &lsquo;Basic Settings&rsquo;.
</p>
<div class="example">
<pre class="example"><a href="https://spack.readthedocs.io/en/latest/configuration.html">https://spack.readthedocs.io/en/latest/configuration.html</a>
<a href="https://spack.readthedocs.io/en/latest/config_yaml.html">https://spack.readthedocs.io/en/latest/config_yaml.html</a>
</pre></div>


<a name="Packages_002eyaml"></a>
<h3 class="section">5 Packages.yaml</h3>

<p>The <code>packages.yaml</code> file specifies the versions and variants for
the packages that spack installs and serves as a common reference
point for HPCToolkit&rsquo;s prerequisites.  This file also specifies the
paths or modules for system build tools (cmake, python, etc) to avoid
rebuilding them.  Put this file in the same directory as
<code>config.yaml</code>.  A sample <code>packages.yaml</code> file is available
in the <code>spack</code> directory of the hpctoolkit repository.
</p>
<p>There are two main sections to <code>packages.yaml</code>.  The first
specifies the versions and variants for hpctoolkit&rsquo;s prereqs.  By
default, spack will choose the latest version of each package (plus
any constraints from hpctoolkit&rsquo;s <code>package.py</code> file).  In most
cases, this will work, but not always.  If you need to specify a
different version or variant, then set this in <code>packages.yaml</code>.
</p>
<p>Note: the versions and variants specified in hpctoolkit&rsquo;s
<code>package.py</code> file are hard constraints and should not be changed.
Variants in <code>packages.yaml</code> are preferences that may be modified
for your local system.  (But don&rsquo;t report a bug until you have first
tried the versions from <code>packages.yaml</code> that we supply.)
</p>
<p>There are at least two packages with a variant that you may need to
change depending on your system.  But always check the current
<code>packages.yaml</code> file to see if any more have been added.
</p>
<ol>
<li> <code>intel-tbb</code> &ndash; for very old Intel or AMD systems that don&rsquo;t support
transactional memory, change <code>+tm</code> to <code>~tm</code>.  (This option has
no effect on non-x86 systems.)

</li><li> <code>libmonitor</code> &ndash; on Blue Gene/Q, add <code>+bgq</code>.
</li></ol>

<p>The second section in <code>packages.yaml</code> specifies a path or module
for system build tools.  Building hpctoolkit&rsquo;s prerequisites requires
cmake 3.4 or later, perl 5.x and python 2.7 or 3.4 or later.  There are
three ways to satisfy these requirements: a system installed version
(eg, /usr), a pre-built module or build from scratch.
</p>
<p>By default, spack will rebuild these from scratch, even if your local
version is perfectly fine.  If you already have an installed version
and prefer to use that instead, then you can specify this in
<code>packages.yaml</code>.  Note that these are only build tools.
Hpctoolkit does not link with any of their libraries.
</p>
<p>For example, this entry says that cmake 3.7.2 is available from module
<code>CMake/3.7.2</code> and that spack should use this instead of building
its own copy.
</p>
<div class="example">
<pre class="example">cmake:
  modules:
    cmake@3.7.2:  CMake/3.7.2
  buildable: False
</pre></div>

<p>This example says that perl v5.16.3 is installed at
<code>/usr/bin/perl</code>.  Note that the <code>paths:</code> entry is the parent
directory of <code>bin</code>, not the bin directory itself (similar to
prefix).
</p>
<div class="example">
<pre class="example">perl:
  paths:
    perl@5.16.3:  /usr
  buildable: False
</pre></div>

<p>See the spack docs on &rsquo;Build Customization&rsquo; and &rsquo;Specs and
Dependencies&rsquo;.
</p>
<div class="example">
<pre class="example"><a href="https://spack.readthedocs.io/en/latest/build_settings.html">https://spack.readthedocs.io/en/latest/build_settings.html</a>
<a href="https://spack.readthedocs.io/en/latest/basic_usage.html#specs-dependencies">https://spack.readthedocs.io/en/latest/basic_usage.html#specs-dependencies</a>
</pre></div>


<a name="Compilers-and-compilers_002eyaml"></a>
<h3 class="section">6 Compilers and compilers.yaml</h3>

<p>Building HPCToolkit requires GNU gcc/g++ at a minimum version 4.8 and
preferably 5.x or later.  By default, spack uses the latest available
version of gcc, but you can specify a different compiler, if one is
available.
</p>
<p>Spack uses a separate file, <code>compilers.yaml</code> to store information
about available compilers.  This file is normally in your home directory
at <code>~/.spack/platform</code> where &lsquo;platform&rsquo; is normally &lsquo;linux&rsquo; (or
else &lsquo;cray&rsquo; or &lsquo;bgq&rsquo;).
</p>
<p>The first time you use spack, or after adding a new compiler, you should
run <code>spack compiler find</code> to have spack search your system for
available compilers.  If a compiler is provided as a module, then you
should load the module before running <code>find</code>.  Normally, you only
need to run <code>find</code> once, unless you want to add or delete a
compiler.  You can also run <code>spack compiler list</code> and <code>spack
compiler info</code> to see what compilers spack knows about.
</p>
<p>For example, on one power8 system running RedHat 7.3, /usr/bin/gcc is
version 4.8.5, but gcc 6.4.0 is available as module <code>GCC/6.4.0</code>.
</p>
<div class="example">
<pre class="example">module load GCC/6.4.0

spack compiler find
==&gt; Added 2 new compilers to /home/krentel/.spack/linux/compilers.yaml
    gcc@6.4.0  gcc@4.8.5
==&gt; Compilers are defined in the following files:
    /home/krentel/.spack/linux/compilers.yaml

spack compiler list
==&gt; Available compilers
-- gcc rhel7-ppc64le --------------------------------------------
gcc@6.4.0  gcc@4.8.5

spack compiler info gcc@6.4
gcc@6.4.0:
    paths:
        cc = /opt/apps/software/Core/GCCcore/6.4.0/bin/gcc
        cxx = /opt/apps/software/Core/GCCcore/6.4.0/bin/g++
        f77 = /opt/apps/software/Core/GCCcore/6.4.0/bin/gfortran
        fc = /opt/apps/software/Core/GCCcore/6.4.0/bin/gfortran
    modules  = ['GCC/6.4.0']
    operating system  = rhel7
</pre></div>

<p>Note: for compilers from modules, spack does not fill in the
<code>modules:</code> field in the <code>compilers.yaml</code> file.  You need to
do this manually.  In the above example, after running <code>find</code>, I
edited <code>compilers.yaml</code> to add <code>GCC/6.4.0</code> to the
<code>modules:</code> field as below.  This is important to how spack
manipulates the build environment.
</p>
<div class="example">
<pre class="example">- compiler:
    modules: [GCC/6.4.0]
    operating_system: rhel7
    spec: gcc@6.4.0
    ...
</pre></div>

<p>Spack uses <code>%</code> syntax to specify the build compiler and <code>@</code>
syntax to specify the version.  For example, suppose you had gcc
versions 7.3.1, 6.4.0 and 5.4.0 available and you wanted to use 6.4.0.
You could write this as:
</p>
<div class="example">
<pre class="example">spack install package %gcc@6.4.0
</pre></div>

<p>See the spack docs on &lsquo;Compiler Configuration&rsquo;.
</p>
<div class="example">
<pre class="example"><a href="https://spack.readthedocs.io/en/latest/getting_started.html#compiler-configuration">https://spack.readthedocs.io/en/latest/getting_started.html#compiler-configuration</a>
</pre></div>


<a name="Spack-Install"></a>
<h3 class="section">7 Spack Install</h3>

<p>First, make sure that you have your <code>config.yaml</code>,
<code>packages.yaml</code> and <code>compilers.yaml</code> files in place and
edited for your system.  You can see how spack will build hpctoolkit
with <code>spack spec</code> and <code>spack graph</code>.
</p>
<div class="example">
<pre class="example">spack spec hpctoolkit
spack graph hpctoolkit
</pre></div>

<p>Then, there are two ways to install hpctoolkit.  The &lsquo;one button&rsquo; method
uses spack to install everything.
</p>
<div class="example">
<pre class="example">spack install hpctoolkit
</pre></div>

<p>Alternatively, especially for developers, you can use spack to install
hpctoolkit&rsquo;s dependencies and then build hpctoolkit with the traditional
<code>configure ; make ; make install</code> method.  Of course, if you
already used the one-button method, then the spack install tree also
contains all of hpctoolkit&rsquo;s prereqs.
</p>
<div class="example">
<pre class="example">spack install --only dependencies hpctoolkit
</pre></div>

<p>Then, run hpctoolkit configure with <code>--with-spack</code> instead of
<code>--with-externals</code>.  The <code>--with-spack</code> option passes the
prerequisite paths to hpctoolkit in place of all of the individual
<code>--with-pkg</code> options (as did externals).
</p>
<p>Spack installs its packages in subdirectories of <code>install_tree</code>
named by architecture (platform, OS, machine type) and compiler
(family, version).  The argument to <code>--with-spack</code> should be the
directory containing all of the individual install directories
(normally two directories down from the install root).  For example,
on my Fedora 26 build machine, I would use:
</p>
<div class="example">
<pre class="example">configure  \
   --prefix=/path/to/hpctoolkit/install/prefix  \
   --with-spack=/path/to/spack/install_tree/linux-fedora26-x86_64/gcc-7.3.1  \
   ...
make -j &lt;num&gt;
make install
</pre></div>

<p>Note: if your spack install tree has multiple versions or variants for
the same package, then <code>--with-spack</code> will select the most recent
one by directory time stamp (and issue a warning).  If this is not
what you want, then you will need to specify the correct version with
a <code>--with-pkg</code> option.
</p>

<a name="Advanced-Options"></a>
<h3 class="section">8 Advanced Options</h3>

<a name="MPI"></a>
<h4 class="subsection">8.1 MPI</h4>

<p>HPCToolkit always supports profiling MPI applications.  For
hpctoolkit, the spack variant <code>+mpi</code> is for building hpcprof-mpi,
the MPI version of hpcprof.  If you want to build hpcprof-mpi, then
you need to supply an installation of MPI.
</p>
<p>Normally, for systems with compute nodes, you should use an existing
MPI module that was built for the correct interconnect for your system
and add this to <code>packages.yaml</code>.  The MPI module should be built
with the same version of GNU gcc/g++ used to build hpctoolkit (to keep
the C++ libraries in sync).
</p>
<a name="PAPI-vs-Perfmon"></a>
<h4 class="subsection">8.2 PAPI vs Perfmon</h4>

<p>HPCToolkit can access the Hardware Performance Counters with either
PAPI or Perfmon (libpfm4).  By default, the hpctoolkit package uses
perfmon.  If you want to use PAPI instead, then build hpctoolkit with
<code>+papi</code>.  However, you can&rsquo;t use both due to a potential conflict
in their header files.
</p>
<p>PAPI runs on top of the perfmon library, but PAPI uses its own,
internal copy of perfmon.  Prior to version 5.6.0, PAPI did not
install the perfmon header files, so it was impossible to access the
perfmon events through PAPI.
</p>
<p>However, starting with version 5.6.0, PAPI now installs both the
perfmon library and its header files.  Hpctoolkit configure will
automatically detect this, so if you build hpctoolkit with a recent
enough version of PAPI, then both the PAPI and perfmon interfaces will
be available.
</p>

<a name="Platform-Specific-Notes"></a>
<h3 class="section">9 Platform Specific Notes</h3>

<a name="Blue-Gene"></a>
<h4 class="subsection">9.1 Blue Gene</h4>

<p>Blue Gene systems are being phased out, without a next generation
replacement, but are still supported by hpctoolkit (for now).  Blue
Gene normally comes with a Red Hat Enterprise Linux (RHEL) 6.x front
end and the GNU gcc/g++ 4.4 compilers with are too old to support the
latest version of hpctoolkit.
</p>
<p>Spack supports two architecture types on Blue Gene.  The front-end arch
is <code>bgq-rhel6-ppc64</code> and the back-end arch is <code>bgq-cnk-ppc64</code>.
Normally, it would matter whether we built for the front end or back
end.  But for the GNU compilers on Blue Gene, both compilers are
/usr/bin/gcc.  So, we just build for the default back end.
</p>
<p>There are two ways to build hpctoolkit for Blue Gene, depending on the
version of the compiler and Dyninst.  The simple method uses the default
GCC 4.4 compiler and Dyninst 9.3.2.  Edit these two entries in the
<code>packages.yaml</code> file.
</p>
<ol>
<li> Set the dyninst version to <code>9.3.2</code>.

</li><li> Add <code>+bgq</code> to the libmonitor variants.
</li></ol>

<p>Then, build hpctoolkit with <code>+bgq</code> turned on.  The <code>+bgq</code>
variant adds MPI and hpcprof-mpi and supersedes the <code>+mpi</code>
variant.
</p>
<div class="example">
<pre class="example">spack install hpctoolkit +bgq
</pre></div>

<p>The advanced method involves building a new compiler (gcc 4.8 or 4.9)
but allows using Dyninst 10.x.  This method requires TCL or Lmod
modules and an MPI C++ compiler built for a compatible version of g++.
Mira and vesta at ANL have softenv modules for +mpiwrapper-gcc and
+bgqtoolchain-gcc484 but don&rsquo;t provide TCL or Lmod modules.
</p>
<p>First, use spack to build a new compiler.  Dyninst 10.x requires
gcc&nbsp;<!-- /@w -->4.8 or later, but libmonitor requires the true back-end
<code>powerpc64-bgq-linux-gcc</code> compiler which is version 4.4.  We
settle on gcc 4.8.5 as a compromise between these two constraints.
Edit <code>config.yaml</code> to set <code>module_roots</code> to a directory in
which to install the modules and then build gcc.
</p>
<div class="example">
<pre class="example">spack install gcc @4.8.5
</pre></div>

<p>Assuming you have modules installed, then load the new gcc module and
rerun <code>spack compiler find</code>.  This should add the new compiler to
your <code>compilers.yaml</code> file (twice, for front and back end).  For
example, on vulcan at LLNL,
</p>
<div class="example">
<pre class="example">module use /path/to/modules/bgq-cnk-ppc64
module load gcc-4.8.5-gcc-4.4.7-nzvjwva
spack compiler find
==&gt; Added 2 new compilers to /g/g21/krentel1/.spack/bgq/compilers.yaml
    gcc@4.8.5  gcc@4.8.5
</pre></div>

<p>Note: after running <code>spack compiler find</code>, you still need to add
the modules field to the compiler entry in <code>compilers.yaml</code> as in
the section on Compilers.
</p>
<p>Note: in the above example, we used gcc 4.4.7 to build gcc 4.8.5 (so
the gcc-4.8.5 package is in the gcc-4.4.7 directory).  You could then
use gcc 4.8.5 to rebuild 4.8.5, if you like, but this is not
necessary.
</p>
<p>Finally, use the new compiler to build hpctoolkit with the new compiler.
If you have modified the dyninst entry in <code>packages.yaml</code>, then
reset the dyninst version to 10.1.0 (or later).
</p>
<div class="example">
<pre class="example">spack install hpctoolkit +bgq %gcc@4.8.5
</pre></div>

<p>If you don&rsquo;t have modules installed, then use <code>spack bootstrap</code>
to build the environment-modules package.  Then, source the bash or
csh script in the <code>Modules/init</code> directory to add the
<code>module</code> function to your environment.  For example,
</p>
<div class="example">
<pre class="example">spack bootstrap
cd /path/to/environment-modules-4.2.4-rwtvlanuss35fxd3xuyldbfopt2m4ecs/init
(bash)  . ./bash
(csh)   source ./csh
</pre></div>

<p>Alternatively, you could try adding the module environment variables
(PATH and LD_LIBRARY_PATH) manually, but then you may have to run
spack install with <code>--dirty</code>.  The <code>--dirty</code> option tells
spack not to erase LD_LIBRARY_PATH while building packages.
</p>
<div class="example">
<pre class="example">spack install --dirty hpctoolkit +bgq %gcc@4.8.5
</pre></div>

<p>For developers, if you are building hpctoolkit directly (outside of
spack) but using spack prerequisites, then use a configure line
similar to the following.
</p>
<div class="example">
<pre class="example">configure  \
    --prefix=/path/to/install/prefix  \
    --with-spack=/path/to/bgq-cnk-ppc64/gcc-4.8.5  \
    --enable-bgq  \
    --enable-all-static  \
    MPICXX=mpicxx
</pre></div>


<a name="Cray"></a>
<h4 class="subsection">9.2 Cray</h4>

<p>Cray systems, like Blue Gene, have separate front and back-end
architecture types.  For example, on theta at ANL, the front-end arch
is <code>cray-sles12-x86_64</code> (SuSE Linux version 12 for x86_64) and
the back-end is <code>cray-cnl6-mic_knl</code> (Compute Node Linux for KNL).
</p>
<p>Hpctoolkit needs to be built with the GNU Programming Environment and
the front-end x86_64 compilers, plus the <code>CC</code> MPI C++ wrapper.
Switch to the <code>PrgEnv-gnu</code> module and unload the darshan module.
Darshan is a profiling tool that monitors an application&rsquo;s use of I/O,
but it conflicts with hpctoolkit.
</p>
<div class="example">
<pre class="example">module swap PrgEnv-intel PrgEnv-gnu
module unload darshan
</pre></div>

<p>By default, spack probably does not recognize the gcc compiler modules
as front-end compilers.  You have to tell spack to add them
explicitly.  Check that the latest gcc module is loaded (it probably
is from <code>PrgEnv-gnu</code>) and run <code>spack compiler find</code> with the
path to the compiler (one directory up from <code>bin</code>).  For example,
on theta at ANL,
</p>
<div class="example">
<pre class="example">module load gcc/7.3.0
spack compiler find /opt/gcc/7.3.0
spack compiler list
==&gt; Available compilers
-- gcc cnl6-any -------------------------------------------------
gcc@7.3.0  gcc@7.2.0  gcc@7.1.0  gcc@6.3.0  gcc@6.1.0  gcc@5.3.0
gcc@4.9.3

-- gcc sles12-x86_64 --------------------------------------------
gcc@7.3.0  gcc@4.8
</pre></div>

<p>Normally, spack does not fill in the <code>modules:</code> field for the new
compiler, you have to add that manually.  And although we mostly use
the front-end environment, we also need the <code>CC</code> wrapper to be
the back-end MPI compiler.
</p>
<p>Spack does not support using both front-end and back-end compilers in
the same build.  But there are two workarounds for Cray.  One option is
to load the <code>cray-mpich</code> module and run <code>spack install</code> with
option <code>--dirty</code>.  A better way is to add <code>cray-mpich</code> to the
list of modules for gcc in <code>compilers.yaml</code>.  Edit the front-end
gcc entry (operating system sles12 and target x86_64) to add the
following modules (your versions may differ).
</p>
<div class="example">
<pre class="example">- compiler:
    modules:
    - PrgEnv-gnu/6.0.4
    - gcc/7.3.0
    - cray-mpich/7.7.3
    operating_system: sles12
    paths:
      cc: /opt/gcc/7.3.0/bin/gcc
      cxx: /opt/gcc/7.3.0/bin/g++
      f77: /opt/gcc/7.3.0/bin/gfortran
      fc: /opt/gcc/7.3.0/bin/gfortran
    spec: gcc@7.3.0
    target: x86_64
</pre></div>

<p>This tells spack to load the above modules when using the gcc 7.3.0
compiler.  With the cray-mpich module loaded, the <code>CC</code> wrapper
becomes an MPI C++ compiler, which is what hpctoolkit needs.
</p>
<p>Next, review your <code>packages.yaml</code> file.  On Cray systems with
Xeon Phi back-end nodes (KNL, KNH, etc), add <code>~tm</code> to intel-tbb
to disable transactional memory.
</p>
<p>Finally, build hpctoolkit with the front-end arch type
(cray-sles12-x86_64) and option <code>+cray</code>.  Normally, you can use OS
type <code>fe</code> (front-end) in place of <code>sles12</code>.  As with Blue
Gene, the <code>+cray</code> option adds MPI and hpcprof-mpi and supersedes
the <code>+mpi</code> variant.
</p>
<div class="example">
<pre class="example">spack install hpctoolkit +cray arch=cray-fe-x86_64
</pre></div>

<p>For developers, if you are building hpctoolkit directly (outside of
spack) but using spack prerequisites, then use a configure line
similar to the following.
</p>
<div class="example">
<pre class="example">configure  \
    --prefix=/path/to/install/prefix  \
    --with-spack=/path/to/cray-sles12-x86_64/gcc-7.3.0  \
    --enable-all-static  \
    MPICXX=CC
</pre></div>


<a name="Hpcviewer-and-Hpctraceviewer"></a>
<h3 class="section">10 Hpcviewer and Hpctraceviewer</h3>

<p>We provide binary distributions for hpcviewer and hpctraceviewer on
Linux (x86_64 and powerpc), Windows and MacOS.  HPCToolkit databases are
platform-independent and it is common to run hpcrun on one machine and
then view the results on another machine.
</p>
<p>All versions of the viewers require Java version 8 (not 9 or later).
The Linux versions also require GTK+ version 2.x.
</p>
<a name="Spack-Install-1"></a>
<h4 class="subsection">10.1 Spack Install</h4>

<p>The spack install is available on Linux x86_64, big-endian power7
(ppc64) and little-endian power8 and 9 (ppc64le).  This installs both
hpcviewer and hpctraceviewer and includes the Java 8 prerequisite.
</p>
<p>On x86_64, use the following command.  Hpcviewer requires jdk 1.8, but
the latest 1.8 version, 1.8.0_202 does not download properly due to
licensing issues, however 1.8.0_141-b15 seems to work.
</p>
<div class="example">
<pre class="example">spack install hpcviewer ^jdk@1.8.0_141-b15
</pre></div>

<p>On powerpc (big and little-endian), use the IBM version of Java.
</p>
<div class="example">
<pre class="example">spack install hpcviewer ^ibm-java
</pre></div>

<a name="Manual-Install"></a>
<h4 class="subsection">10.2 Manual Install</h4>

<p>Binary distributions of the viewers for all supported platforms are
available at:
</p>
<div class="example">
<pre class="example"><a href="http://hpctoolkit.org/download/hpcviewer">http://hpctoolkit.org/download/hpcviewer</a>
</pre></div>

<p>On Linux, download the <code>linux.gtk</code> versions of hpcviewer and
hpctraceviewer, unpack the tar files and run the install scripts (for
both viewers) with the path to the desired install prefix.
</p>
<div class="example">
<pre class="example">./install /path/to/install/directory
</pre></div>

<p>On Windows and MacOS, download the <code>win32</code> or <code>macosx.cocoa</code>
versions and unpack the zip files in the desired directory.  Due to
Apple&rsquo;s security precautions, on MacOS, you will need to use curl or
wget instead of a web browser.
</p>

<a name="Common-Problems"></a>
<h3 class="section">11 Common Problems</h3>

<a name="Unable-to-fetch-tar-file"></a>
<h4 class="subsection">11.1 Unable to fetch tar file</h4>

<p>Spack is somewhat fragile for how it downloads tar files and will
often fail for transitory network problems.  This is especially true
for packages with many dependencies.  For example:
</p>
<div class="example">
<pre class="example">==&gt; Installing m4
==&gt; Searching for binary cache of m4
==&gt; No binary for m4 found: installing from source
curl: (6) Could not resolve host: ftp.wayne.edu; Name or service not known
==&gt; Fetching https://ftpmirror.gnu.org/m4/m4-1.4.18.tar.gz
==&gt; Fetching from https://ftpmirror.gnu.org/m4/m4-1.4.18.tar.gz failed.
==&gt; Error: FetchError: All fetchers failed for m4-1.4.18-vorbvkcjfac43b7vuswsvnm6xe7w7or5
</pre></div>

<p>There are two workarounds.  First, assuming the problem is temporary,
simply wait 10 minutes or an hour and try again.
</p>
<p>Second, you could set up a spack mirror.  A mirror allows you to
download a tar file ahead of time and put it in a directory where
spack can find it without having to fetch it over a network.  A mirror
is also useful for files with license issues (eg, jdk 1.8.0_202 as
above), or on a secure machine without direct access to the internet.
See:
</p>
<div class="example">
<pre class="example"><a href="https://spack.readthedocs.io/en/latest/mirrors.html">https://spack.readthedocs.io/en/latest/mirrors.html</a>
</pre></div>

<a name="New-releases-break-the-build"></a>
<h4 class="subsection">11.2 New releases break the build</h4>

<p>Normally, HPCToolkit should build and work correctly with the latest
version for all of its dependencies.  But sometimes a new release will
change something and break the build.  This has happened a couple times
where a new release of Boost has broken the build for Dyninst.  Or,
maybe the latest version of gcc/g++ disallows some usage and breaks the
build.
</p>
<p>The solution is to use <code>packages.yaml</code> to specify an earlier
version until the rest of the code adapts to the change.
</p>
<hr>



<div style="width: 100%; font-size: small;">
  <hr />
  <p style="margin: 0px; font-size: small;">
    &copy;2000-2018 <a href="http://www.rice.edu">Rice University</a>
    &bull;
    <a href="http://www.cs.rice.edu">Rice Computer Science</a>
  </p>
  <a href="http://validator.w3.org/check/referer">
    <img src="http://www.w3.org/Icons/valid-xhtml10-blue" alt="" height="15" /></a>
  <a href="http://jigsaw.w3.org/css-validator/check/referer">
    <img src="http://www.w3.org/Icons/valid-css-blue" alt="" height="15" /></a>
</div>

</body>
</html>
